Learning a context-free task with a recurrent neural network: An analysis of stability

Natural languages exhibit context-free properties such as center-embedded clauses. Recent research has sought a model that performs on these features with human-like inconsistencies, rather than like traditional discrete automata. This search has recently focussed on recurrent neural networks. It has been shown theoretically that recurrent networks are computationally as powerful as Turing machines (Siegelmann, 1993). However, the class of problems that recurrent networks can learn, and the solutions that the networks nd are still not known. Rodriguez, Wiles and Elman (1996) looked at the representation employed by a recurrent neural network to process the context-free language a n b n. They found that highly coordinated dynamical structures were required for the network to perform the task. This paper examines how a recurrent network learns these dynamics given the structural dependencies required. To observe the evolution of the networks, their weights and performance were monitored during training. We found that as a network approached the solution, the weights became unstable and any solution that was found was soon lost. An analysis of the changes in dynamics as the network loses a solution indicates that small changes in weights result in signiicant changes to the dynamics of the network. We conclude that learning context-free languages may be inherently unstable, but that further work is necessary to explore the possible reasons for this instability.